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PREFACE 


The  work  reported  here  began  in  connection  with  the  ROSIE® Language  Development  Project,  which 
was  funded  by  the  Information  Sciences  and  Technologies  Office  of  the  Defense  Advanced  Research 
Projects  Agency  under  the  auspices  of  The  RAND  Corporation’s  National  Defense  Research  Insti¬ 
tute,  a  Federally  Funded  Research  and  Development  Center  sponsored  by  the  Office  of  the  Secretary 
of  Defense.  Since  the  completion  of  the  ROSIE  project,  this  work  was  continued  independently 
by  the  author.  This  Paper  is  a  companion  to  the  RAND  Note  “A  TABLE-DRIVEN  APPROACH 
TO  FAST  CONTEXT-FREE  PARSING”  (N-2841-DARPA).  The  intention  in  publishing  both  doc¬ 
uments  is  to  make  the  parsing  techniques  applied  in  ROSIE  available  for  use  by  others  at  RAND 
and  elsewhere. 
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SUMMARY 


A  variation  on  Tomita’s  algorithm  for  general  context-free  parsing  is  analyzed  in  regards  to  its 
time  complexity.  It  is  shown  to  have  a  general  time  bound  proportional  to  n^+1,  where  n  is  the 
length  of  the  input  string  and  p  is  the  length  of  the  longest  production  in  the  source  grammar. 
A  modification  of  this  algorithm  is  presented  for  which  this  time  bound  is  reduced  to  a  factor  of 
n3.  A  discussion  of  space  bounds,  as  well  as  two  subclasses  of  context-free  grammars  that  can  be 
recognized  in  less  than  0(n3)  time,  is  also  included. 
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1.  INTRODUCTION 


Context-free  grammars  (Chomsky,  1956)  have  been  widely  used  in  describing  the  syntax  of 
programming  and  natural  languages.  Numerous  algorithms  have  been  developed  to  recognize  sen¬ 
tences  in  languages  so  described.  Some  are  general,  in  the  sense  that  they  can  handle  all  or  most 
context-free  grammars;  others  are  more  restricted  and  can  handle  only  a  small  subclass  of  these 
grammars  (including  the  grammars  of  most  programming  languages).  These  latter  algorithms,  e.g., 
the  LL,  operator  precedence,  predictive,  and  LR  parsing  algorithms  (Aho  and  Ullman,  1972)  are 
typically  more  efficient  than  the  former  because  they  take  advantage  of  inherent  features  in  the  class 
of  grammars  they  recognize. 

Most  practical  parsers  analyze  the  syntax  of  their  input  in  a  single  deterministic  pass,  without 
need  of  backup.  Each  symbol  is  examined  only  once,  and,  at  the  time  it  is  examined,  there  is  sufficient 
information  available  to  make  any  necessary  parsing  decisions.  In  his  famous  paper,  Knuth  (Knuth, 
1965)  established  a  family  of  context-free  grammars  known  as  LR(k)  grammars  and  provided  an 
effective  test  to  determine,  for  a  given  positive  integer  k,  whether  a  grammar  belonged  to  the  LR(lt) 
class.  The  connection  to  practical  parsers  mentioned  above  is  that  an  LR(it)  grammar  describes  a 
language,  all  of  whose  sentences  can  be  parsed  in  a  single  backup-free  parse,  if  at  most  k  symbols 
of  look-ahead  are  available.  Despite  the  wide  coverage  of  LR(&)  grammars,  there  are  still  many 
languages  for  which  no  LR(fc)  grammar  exists.  The  example  that  sparked  this  work  is  ROSIE 
(Kipps  et  al.,  1987),  a  language  for  applications  in  artificial  intelligence  with  a  high-level  English- 
like  syntax. 

The  general  context-free  parsing  algorithms,  e.g.,  Earley’s  algorithm  (Earley,  1968;  1970),  and 
the  Cocke- Younger-Kasaini  algorithm  (Younger,  1967),  must  necessarily  simulate  a  nondeterministic 
pass  over  their  input,  using  some  form  of  search.  Due  to  the  inefficiency  this  causes,  these  algorithms 
have  not  been  widely  used  as  practical  parsers  for  programming  languages.  However,  as  programming 
languages,  like  ROSIE,  begin  to  approach  natural  language  in  readability  and  expressiveness,  they 
will  also  become  ambiguous  and  non-LR(fc)  and,  thus,  inherently  harder  to  recognize.  Such  grammars 
require  the  power  of  a  general  parser.  Although  the  best  time  bound  for  the  general  context-free 
algorithms  is  0(n3),2  where  n  is  the  length  of  the  input  string,  some  of  these  algorithms  run  faster  for 
certain  subclasses  of  context-free  grammars.  For  instance,  in  his  thesis  (Earley,  1968),  Earley  shows 


2  Actually,  the  best  upper  bound  is  Valiant  s  algorithm,  which  runs  in  0(n2  81).  However,  since  this  is 
also  its  lower  bound,  Valiant’s  algorithm  is  only  of  theoretical  interest. 


how  his  algorithm  runs  in  0{n7)  for  unambiguous  grammars  and  in  time  O(n)  for  bounded  state 
grammars.  Unambiguous  and  bounded  state  grammars  subsume  the  LR(fc)  grammars,  and  they  may 
also  subsume  a  large  subset  of  interesting  non-LR(i)  grammars.  Although  Earley’s  algorithm  is  still 
too  inefficient  to  be  used  as  a  practical  parser  for  even  these  grammars,  their  existence  suggests  that 
a  sufficiently  fast  algorithm  for  practical  general  context-free  parsing  could  be  developed. 

A  basic  characteristic  that  seems  to  be  shared  by  the  best  known  of  the  general  context-free 
algorithms  is  that  they  are  top-down  parsers.  Recently,  however,  Tomita  (Tomita,  1985a;b)  intro¬ 
duced  an  algorithm  (intended  for  natural  language  applications)  that  takes  advantage  of  an  extended 
LR  parse  table.  This  would  appear  to  be  the  first  general  context-free  algorithm  that  is  a  bottom-up 
parser.  The  obvious  benefit  of  this  approach  is  that  it  eliminates  the  need  to  expand  alternatives  of 
a  nonterminal  at  parse  time  (i.e.,  what  Earley  calls  the  predictor  operation).  For  Earley’s  algorithm, 
eliminating  the  predictor  operation  would  not  change  its  upper-bound  time  complexity,  but  it  would 
save  a  factor  of  n2,  which  in  itself  could  be  significant  to  a  practical  parser.  Unfortunately,  upon 
examination  Tomita’s  algorithm  is  found  to  have  a  general  time  complexity  of  0(n'!+1),  where  n  is 
as  before  and  p  is  the  length  of  the  longest  production  in  the  source  grammar.  Thus,  this  algorithm 
achieves  D(n3)  for  grammars  in  Chomsky  normal  form  (Chomsky,  1959)  but  has  potential  for  being 
worse  in  unrestricted  grammars. 

In  this  paper,  I  present  a  modification  of  Tomita’s  algorithm  that  allows  it  to  run  in  time 
proportional  to  n3  for  grammars  with  any  length  productions.  First,  in  Section  2,  the  terminology 
used  in  this  paper  is  defined.  A  variantion  on  Tomita’s  algorithm  is  described  informally  as  a 
recognizer  in  Section  3,  and  formally  defined  and  analyzed  in  Section  4.  Section  5  describes  and 
analyzes  the  modification  to  this  algorithm  that  allows  it  to  run  in  time  0(n3).  Section  6  examines 
the  cost  in  terms  of  space  required  by  the  modified  algorithm.  Finally,  Section  7  describes  two 
subclasses  of  context-free  grammars  for  which  the  algorithm  runs  in  0(n2)  and  0(n),  respectively. 


-  3- 

2.  TERMINOLOGY 

A  language  is  a  set  of  strings  over  a  finite  set  of  symbols.  These  symbols  are  called  terminals 
and  are  represented  by  lowercase  letters,  e.g.,  a,  b.  c.  A  context-free  grammar  is  used  as  a  formal 
device  for  specifying  which  strings  are  in  a  language;  hereafter,  grammar  is  used  to  mean  context-free 
grammar.  A  grammar  uses  another  set  of  symbols  called  nonterminals ,  which  define  the  syntactic 
classes  of  the  language;  nonterminals  are  represented  by  capital  letters,  e  g.,  A,  B,  C.  Together  the 
terminals  and  nonterminals  of  a  language  make  up  its  vocabulary.  Strings  of  vocabulary  symbols  are 
represented  by  Greek  letters,  e.g.,  a,  0,  7.  The  empty  string  is  e.  |o|  is  the  number  of  symbols  in  a. 
A  grammar  consists  of  a  finite  set  of  rewrite  rules  or  productions  of  the  form 

A  — «  o 

where  the  ‘A’  component  is  called  the  left-hand  side  of  the  production,  and  the  ‘a’  component  is 
called  its  right-hand  side.  The  nonterminal  that  stands  for  “sentence”  is  called  the  root  ( R )  of  the 
grammar.  Productions  with  the  same  nonterminal  on  their  left-hand  side  are  called  alternatives  of 
that  nonterminal.  Productions  of  the  form 

A  — 1 •  c 

are  called  null  productions. 

The  rest  of  the  definitions  are  given  with  respect  to  a  particular  source  grammar  G.  We  write 

a  =>  0 

if  3‘i,5.  rj,t.  such  t.b-.t  a-  —  ykt  and  0  —  yrjb  and  /.  -  is  a  production  We  write 

a  S*  0 

(0  is  derived  from  a)  if  3ao,»i,  ■  •  -  ,am  (m  >  0)  such  that 

«  =  c*o  =>  c*i  =>  ■  ■  ■  =>  am  =  0. 

The  sequence  a0,  ■  •  • ,  am  is  called  a  derivation  (of  0  from  o). 

A  sentential  form  is  a  string  a  such  that  the  root  R  a.  A  sentence  is  a  sentential  form 
consisting  entirely  of  terminal  symbols.  The  language  defined  by  a  grammar  L(G)  is  the  set  of  its 
sentences.  Any  sentential  form  may  be  represented  in  at  least  one  way  as  a  derivation  tree,  reflecting 
the  steps  made  in  deriving  it  (though  not  the  order  of  the  steps).  The  degree  of  ambiguity  of  a 
sentence  is  the  number  of  its  distinct  derivation  trees.  A  sentence  is  unambiguous  if  it  has  degree  1 
of  ambiguity.  A  grammar  is  unambiguous  if  each  of  its  sentences  is  unambiguous. 

A  recognizer  is  an  algorithm  that  takes  as  its  input  a  string  and  either  accepts  or  rejects  it, 
depending  on  whether  the  string  is  a  sentence  of  the  language  defined  by  the  grammar.  A  parser  is 
a  recognizer  that  outputs  the  set  of  all  legal  derivation  trees  of  a  string  upon  acceptance. 
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3.  TOMITA'S  ALGORITHM 


The  following  is  an  informal  description  of  a  variation  on  Tomita’s  algorithm  as  a  recognizer. 
I  assume  familiarity  with  standard  LR  parsing,  the  exact  definition  and  operation  of  which  can  be 
found  in  Aho  and  Ullman,  1972 .  The  changes  introduced  to  Tomita’s  algorithm  do  not  alter  it 
significantly,  but  they  do  make  it  easier  to  describe  and  analyze.  The  algorithm  described  below  is 
based  on  an  implementation  of  Tomita’s  algorithm  as  a  practical  parser  (Kipps,  1988)  distributed 
with  the  ROSIE  language. 

Tomita  views  his  algorithm  as  a  variation  on  standard  LR  parsing.  The  algorithm  takes  a 
shift-reduce  approach,  using  an  extended  LR  parse  table  to  guide  its  actions.  The  changes  the 
algorithm  makes  to  the  parse  table  are  to  allow  it  to  contain  multiple  actions  per  entry,  i.e.,  at  most 
one  shift  action  or  accept  action  and  any  number  of  reduce  actions.  Thus,  the  parse  table  can  no 
longer  be  used  for  strictly  deterministic  parsing;  some  search  must  be  done.  The  algorit  hm  emulates 
a  nondeterministic  parse  with  pseudo-parallelism.  It  scans  an  input  string  xi  ■  ■  x„  from  left  to 
right,  following  all  paths  in  a  breath-first  manner  and  merging  like  subpaths  when  possible  to  avoid 
redundant  computations. 

An  example  non-LR  grammar  is  shown  in  Figure  3.1, 


(1) 

S 

— 

HP  VP 

(2) 

s 

— * 

S  PP 

(3) 

NP 

— 

♦n 

(4) 

HP 

— * 

♦det  *n 

(5) 

HP 

— 

HP  PP 

(6) 

PP 

— 

♦prep  HP 

(7) 

VP 

— 

♦v  HP 

Fig.  3.1 — Example  of  Non-LR  Grammar 

and  its  parse  table  in  Figure  3.2.  In  building  the  parser  table  the  grammar  is  augmented  by  a  Oth 
production 

D0  -  fH 

where  R  is  the  root  of  the  grammar  and  where  the  symbol  H’  is  a  special  terminal  that  denotes 
end-of-sentence  and  appears  only  as  the  last  symbol  of  an  input  string.  Entries  ‘sh  s’  in  the  action 
table  (the  left  part  of  the  table)  indicate  the  action  ‘ shift  to  State  s.'  Entries  ‘re  p'  indicate  the 
action  ‘reduce  constituents  on  the  stack  according  to  Production  p.’  The  entry  ‘acc’  indicates  the 
action  'accept,'  and  blank  spaces  represent  'error.'  Entries  's'  in  the  goto  table  (the  right  part  of 
the  table)  indicate  the  action  after  a  reduce  action,  shift  to  State  s.'  In  Figure  3.2,  there  are  two 
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multi-action  entries  in  States  3  and  11  under  the  column  labeled  *prep  ' 


State 

♦det 

*n 

*v 

♦prep 

H 

HP 

PP 

VP 

S 

0 

sh5 

sill 

9 

1 

1 

sh2 

acc 

8 

2 

sh5 

sill 

3 

3 

re6 

re6,sh2 

red 

4 

4 

re5 

re5 

re5 

5 

sh6 

6 

r«4 

re4 

re4 

7 

re3 

re3 

r«3 

8 

re2 

re2 

9 

shlO 

sh2 

4 

12 

10 

sh5 

sh7 

11 

11 

re7,sh2 

re7 

4 

12 

rel 

rel 

Fig.  3.2 — LR  Parse  Table  with  Multiple  Entries 

The  algorithm  operates  by  maintaining  a  number  of  processes  in  parallel.  Each  process  has  a 
stack  and  behaves  basically  the  same  as  in  standard  LR  parsing.  Each  stack  element  is  labeled  with 
a  parse  state  and  points  to  its  parent,  i.e.,  the  previous  element  on  a  process’s  stack.  We  call  the 
top-of-stack  the  current  slate  of  a  process. 

Each  process  does  not  actually  maintain  its  own  separate  stack.  Rather,  these  “multiple”  stacks 
are  represented  using  a  single  directed  acyclic  (but  reentrant)  graph  called  a  graph-structured  stack. 
Each  stack  element  is  a  vertex  of  the  graph.  Each  leaf  of  the  graph  acts  as  a  distinct  top-of-stack 
to  a  process,  while  the  root  of  the  graph  acts  as  a  common  bottom-of-stack.  The  edge  between  a 
vertex  and  its  parent  is  directed  toward  the  parent.  Because  of  the  reentrant  nature  of  the  graph,  a 
vertex  may  have  more  than  one  parent. 

The  leaves  of  the  graph  grow  in  stages;  each  stage  Ui  corresponds  to  the  ith  symbol  x*  from 
the  input  string.  After  x,  is  scanned,  the  leaves  in  stage  f/»  are  in  a  one-to-one  correspondence  with 
the  algorithm’s  active  processes,  where  each  process  references  a  distinct  leaf  and  treats  that  leaf  as 
its  current  state.  Upon  scanning  Xj+i,  an  active  process  can  either  (1)  add  another  leaf  to  Ui,  or 
(2)  add  a  new  leaf  to  Ui+i.  Only  processes  that  added  leaves  to  {/<+ 1  will  be  active  when  Xi+2  is 
scanned. 

In  general,  a  process  behaves  in  the  following  manner  On  x,-,  each  active  process  (corresponding 
to  the  leaves  in  Ui- 1)  looks  up  and  executes  the  entries  in  the  action  table  for  x,  given  its  current 
state.  When  a  process  encounters  multiple  actions,  it  splits  into  several  processes  (one  for  each 
action),  each  sharing  a  common  top-of-stack.  When  a  process  encounters  an  error  entry,  the  process 
is  discarded  (i.e.,  its  top-of-stack  vertex  sprouts  no  leaves  into  Ui  by  way  of  that  process).  Al! 
processes  are  synchronized,  scanning  the  same  symbol  at  the  same  time.  Thus,  after  a  process  shifts 
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on  Xj  into  Ui,  it  waits  until  there  is  no  other  processes  that  can  act  on  x,  before  scanning  x;  +  i. 

The  Shift  Action.  A  process  (with  top-of-stack  vertex  t<)  shifts  on  x,  from  its  current  state  s  to 
some  successor  state  s'  by 

(1)  creating  a  new  leaf  v'  in  U,  labeled  s'; 

(2)  placing  an  edge  from  v'  to  its  top-of-stack  v  (directed  towards  v);  and 

(3)  making  v1  its  new  top-of-stack  vertex  (in  this  way  changing  its  current  state). 

Any  s>’ccessive  process  shifting  to  the  same  state  s'  in  U,  is  merged  with  the  existing  process  to  form 
a  single  process  whose  top-of-stack  vertex  has  multiple  parents,  i.e.,  by  placing  an  additional  edge 
from  the  top-of-stack  vertex  of  the  existing  process  in  Ui  to  the  top-of-stack  vertex  of  the  shifting 
process.  The  merge  is  done  because,  individually,  these  processes  would  behave  in  exactly  the  same 
manner  until  a  reduce  action  removed  the  vertices  labeled  s'  from  their  stacks.  Thus,  merging  avoids 
redundant  computation.  Merging  also  insures  that  each  leaf  in  any  Ui  will  be  labeled  with  a  distinct 
parse  state,  which  puts  a  finite  upper-bound  on  the  possible  number  of  active  processes  and,  thus, 
the  size  of  the  graph-structured  stack. 

The  Reduce  Action.  A  process  executes  a  reduce  action  on  a  production  p  by  following  the  chain 
of  parent  links  down  from  its  top-of-stack  vertex  v  to  the  ancestor  vertex  from  which  the  process 
began  scannin6  for  p  earlier,  essentially  “popping’-  intervening  vertices  off  its  stack.  Since  merging 
means  a  vertex  can  have  multiple  parents,  the  reduce  operation  can  lead  back  to  multiple  ancestors. 
When  this  happens,  the  process  is  again  split  into  separate  processes  (one  for  each  ancestor).  The 
ancestors  will  correspond  to  the  set  of  vertices  at  a  distance  p  from  u,  where  p  equals  the  number  of 
symbols  in  the  right-hand  side  of  the  pth  production.  Once  reduced  to  an  ancestor,  a  process  shifts 
to  the  state  s'  indicated  in  the  goto  table  for  Dp  (the  nonterminal  on  the  left-hand  side  of  the  pth 
production)  given  the  ancestor’s  state.  A  process  shifts  on  a  nonterminal  much  as  it  does  a  terminal, 
with  the  exception  that  the  new  leaf  is  added  to  Ui- 1  rather  than  Ui .  (A  process  can  only  enter  Ui 
by  shifting  on  x,  .) 

The  algorithm  begins  with  a  single  initial  process  whose  top-of-stack  vertex  is  the  root  of  the 
graph-structured  stack.  It  then  follows  the  general  procedure  outlined  above  for  each  symbol  in  the 
input  string,  continuing  until  there  are  either  no  leaves  added  to  Ui  (i.e.,  no  more  active  processes), 
which  denotes  rejection,  or  a  process  executes  the  accept  action  on  scanning  the  n+  1st  input  symbol 
‘ H, ’  which  denotes  acceptance . 
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4.  ANALYSIS  OF  TOMITA’S  ALGORITHM 


In  this  section,  I  present  a  formal  definition  of  the  variation  on  Tomita’s  algorithm  described 
in  Section  3,  as  a  recognizer  for  an  input  string  xi  •  •  •  x„  and  then  analyze  the  tiirm  complexity  of 
the  algorithm  according  to  this  definition.  This  definition  is  understood  to  be  with  respect  to  an 
extended  LR  parse  table  (with  start  state  So)  constructed  from  a  source  grammar  G. 

Notation.  Number  the  productions  of  G  arbitrarily  1,  ■  ■  ■,<!,  where  each  production  is  of  the 

form 

Dp  — +  Cp i  •  •  •  Cpp  (I  <  p  <  d) 

and  where  p  is  the  number  of  symbols  on  the  right-hand  side  of  the  pth  production. 

Definition.  The  entries  of  the  extended  LR  parse  table  are  accessed  with  the  functions  ACTIONS 
and  GOTO. 

•  ACTIONS(s,x)  returns  a  set  of  actions  from  the  action  table  along  the  row  of  State  s  under  the 
column  labeled  ‘ x .’  This  set  will  contain  no  more  than  one  of  a  shift  action  ‘sh  sn  or  an  accept 
action  ‘acc;’  it  may  contain  any  number  of  reduce  actions  ‘re  p.’ 

•  GOTO(s,Dp)  returns  a  state  ‘ s ”  from  the  goto  table  along  the  row  of  State  s  under  the  column 
labeled  with  nonterminal  Dp. 

Definition.  Each  vertex  of  the  graph-structured  stack  is  a  triple  ( i,s,l ),  where  i  is  an  integer 
corresponding  to  ith  input  symbol  scanned  (i.e.,  the  time  at  which  the  vertex  was  created  as  a  leaf), 
s  is  a  parse  state  (corresponding  to  a  row  of  the  parse  table),  and  /  is  a  set  of  parent  vertices.  The 
processes  described  in  the  last  section  are  represented  implicitly  by  the  vertices  in  successive  t/j’s. 
The  root  of  the  graph-structured  stack,  and  hence  the  initial  process,  is  the  vertex  (0,So,  {})- 

The  Recognizer.  The  recognizer  is  a  function  of  one  argument  REC(xi  •  •  •  x„).  It  calls  upon 
the  functions  SHIFT(u,s)  and  REDUCE(v,p).  SHIFT(u,s)  either  (1)  adds  a  new  leaf  to  Ui  labeled 
with  parse  state  s  whose  parent  is  vertex  v  or  (2)  merges  vertex  v  with  the  parents  of  an  existing 
leaf;  REDUCE(v,p)  executes  a  reduce  action  from  vertex  v  using  production  p.  REDUCE  calls  upon 
the  function  ANCESTORS(u,p),  which  returns  the  set  of  all  ancestor  vertices  a  distance  of  p  from 
vertex  v.  These  function  are  defined  in  Figure  4.1. 

The  definitions  in  Figure  4.1  vary  somewhat  from  the  formal  definition  given  in  (Tomita, 
1985a).3  As  a  brief  explanation,  in  REC,  [1]  adds  the  end-of-sentence  symbol  ‘T  to  the  end  of 

3  The  changes  introduced  to  Tomita’s  algorithm  do  not  alter  it  significantly,  but  they  do  make  it  easier  to 
describe.  In  particular,  Tomita’s  functions  REDUCE  and  REDUCE-E  have  been  collapsed  into  a  single  func¬ 
tion. 


-8- 


the  input  string;  [2]  initializes  the  root  of  the  graph-structured  stack;  [3]  iterates  through  the  sym¬ 
bols  of  the  input  string.  On  each  symbol  x;,  [4]  processes  the  vertices  (denoting  the  active  processes) 

REC(xj  x„) 

[1]  Let  x„+1  :=  H 

Let  Ui  be  empty  (0  <  i  <  n) 

[23  Let  Uo  :=  {<0,So,{})} 

[3]  For  i  from  1  to  n  -t-  1 

Let  P  be  empty 

[4]  For  each  v  =  (i—l,s,l)  G 

Let  P  :=  P  U  {u} 

[5]  If  ‘sh  s1'  G  ACTI0NS(s , x,) ,  SHIFT(t>,s') 

[6]  For  each  ‘re  p’  G  ACTI0NS(s,xt) ,  REDUCER, p) 

[7]  If  ‘acc’  G  ACTI0NS(s ,Xj) »  accept 
[83  If  U,  is  empty,  reject 

SHIFT(u ,s) 

[93  If  ~3 (»,«,/)  G  Uit 

let  Ui  :=  Ui  U  { (*,  s,  {t>})} 

else 

let  l  :=  l  U  {t} 

REDUCE(u.p) 

[103  For  each  v'  =  (/,«',/ 1')  G  AHCEST0RS(e,p) , 

Let  s"  :=  GOTO  (s',  Z)p) 

[113  If  ~3v"  =  {i—  1,  s",  l")  e  Ui- 1, 

let  Ui- 1  :=  Ui- 1  U  {<i-l,s",{t/})} 

else 

[123  If  v'  <£  l", 

[133  if  ~3(/,s',/2')  G 

let  l"  :=  l"  U  {u'} 

[143  if  v"  G  P, 

let  vd  :=  (i  —  l,s",{t/}) 

for  each  ‘re  p»  G  ACTI0HS(s",x.) , 

REDUCE (td.p) 

else 

do  nothing  (ambiguous) 

ANCESTORS  (e  =  (;,s,/),c) 

[153  If  c  =  0, 

retum({u}) 

else 

return((Jv,€,  ANCESTORS (v',c  —  1)) 

Fig.  4.1 — Tomita’s  Algorithm 

of  successive  U%- i’s,  adding  each  vertex  to  P  to  signify  that  it  has  been  processed;  [5],  [6]  and  [7] 
respectively  execute  the  shift,  reduce  and  accept  actions  from  the  action  table  given  the  state  s  of  a 
vertex;  and  [8]  checks  that  at  least  one  vertex  was  added  to  Ui,  insuring  that  at  least  one  process  is 
still  active  after  processing  x,  and  before  scanning  x,+i . 
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In  SHIFT,  [9]  adds  a  vertex  to  [/,  labeled  s  (i.e.,  shifts  a  process  to  state  s  in  t/,).  If  a  vertex 
labeled  s  does  not  already  exist,  one  is  created  with  a  single  parent  u;  otherwise,  v  is  added  to  the 
parents  of  the  existing  vertex,  thus  merging  processes. 

In  REDUCE,  [10]  iterates  through  the  ancestor  vertices  a  distance  of  p  from  v,  setting  s"  to 
the  state  indicated  in  the  goto  table  on  Dp  given  the  state  of  the  ancestor.  Each  ancestor  vertex  is 
shifted  into  [7j_i  on  s";  [11]  adds  a  vertex  labeled  s"  to  t/j_i  if  no  such  vertex  already  exists.  If  such 
a  vertex  does  exist,  [12]  checks  that  a  shift  from  the  current  ancestor  v'  has  not  already  been  made. 
(If  it  has,  then  some  segment  of  the  input  string  has  been  recognized  as  an  instance  of  the  same 
nonterminal  Dp  in  two  different  ways,  and  the  current  derivation  can  be  discarded  as  ambiguous; 
otherwise,  v'  is  merged  with  the  parents  of  the  existing  vertex.)  Before  merging,  [13]  checks  that  v' 
is  not  a  “dummy”  vertex,  created  by  [14]  from  an  earlier  call  to  REDUCE;  [14]  checks  if  the  vertex 
v"  has  already  been  processed.  If  so,  then  it  missed  any  possible  reductions  through  t/,  so  a  dummy 
vertex  v<i  is  created  as  a  variant  on  v"  with  a  single  parent  v' .  For  all  such  reduce  actions,  REDUCE 
is  called  using  Vd  in  place  of  v".  Because  of  reduce  actions  on  null  productions,  it  is  possible  for 
ANCESTORS  to  return  a  dummy  vertex  as  the  ancestor  of  itself;  so,  going  back  to  [13],  if  a  variant 
of  v'  already  exists  in  the  parents  of  v",  then  v'  is  a  dummy  vertex  and  can  be  discarded;  otherwise, 
it  is  a  real  vertex  and  can  be  added  to  the  parents  of  v" . 

Finally,  in  ANCESTORS,  [15]  recursively  descends  the  chain  of  parents  of  vertex  v,  returning 
the  set  of  vertices  a  distance  of  c  from  v. 

The  changes  introduced  in  Figure  4.1  do  not  alter  Tomita’s  algorithm  significantly,  but  they  do 
make  it  easier  to  develop  some  ideas  about  its  efficiency.  Here  we  wish  to  find  the  upper  bounds  on 
time  as  a  function  of  n  (the  length  of  the  input  string).  Since  there  are  existing  general  context-free 
parsing  algorithms  that  are  0(n3)  with  respect  to  time,  it  is  of  interest  to  analyze  Tomita’s  algorithm 
and  see  how  it  compares. 

The  General  Case.  Tomita’s  algorithm  is  an  0(n?+1)  recognizer  in  general,  where  p  is  the 
greatest  p  in  G.  The  reasons  for  this  are: 

(a)  Since  each  vertex  in  (7<  must  be  labeled  with  a  distinct  parse  state,  the  number  of  vertices  in 
any  17,  is  bounded  by  the  number  of  parse  states; 

(b)  The  number  of  parents  I  of  a  vertex  v  =  ( i ,  s,  l)  in  t/,  is  proportional  to  i  (~  »).  This  is  because 
a  process  could  have  started  scanning  for  a  production  p  in  each  Uj  ( j  <  i )  and,  thus,  a  process 
could  reduce  on  p  in  Ui  and  split  into  to  ~  i  processes  (one  for  each  ancestor  in  a  distinct  Uj). 
Each  process  could  shift  on  Dp  to  the  same  state  in  Ui  and,  thus,  that  vertex  could  have  ~  i 
parents; 

(c)  For  each  Xi+i,  SHIFT  will  be  called  a  bounded  number  of  times,  i.e.,  at  most  once  for  each 
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vertex  in  Ui\  SHIFT  executes  in  a  bounded  number  of  steps. 

(d)  For  each  x*+i  and  production  p,  REDUCE(t>,p)  will  be  called  a  bounded  number  of  times  in 
REC,  and  REDUCER, p)  (the  recursive  call  to  REDUCE)  will  be  called  no  more  than  ~  i 
times.  The  reason  for  the  former  is  the  same  as  in  (c),  while  the  latter  is  due  to  the  conditions 
on  the  recursive  call,  which  maintain  that  it  can  be  called  no  more  than  once  for  each  parent 
of  a  vertex  in  f7< ,  of  which  there  are  at  most  ~  *; 

(e)  REDUCE(v,p),  because  at  most  ~  i  vertices  can  be  returned  by  ANCESTORS,  executes  in  ~  i 
steps  plus  the  steps  needed  to  execute  ANCESTORS. 

(f)  ANCESTORS(u,p)  executes  in  ~  steps  in  the  worst  case.  This  is  because,  while  at  most  ~  i 
processes  could  have  started  scanning  for  p,  the  number  of  paths  by  which  any  single  process 
could  reach  v  in  U,  is  dependent  upon  the  number  of  ways  the  intervening  input  symbols  can 
be  partitioned  among  the  p  vocabulary  symbols  in  the  right-hand  side  of  production  p.  For  a 
process  that  started  from  Uj  (j  <  i),  the  number  of  paths  to  i>  in  Ui  in  the  recognition  of  p  can 
be  proportional  to 

0  0  0 

£  £  £  >• 

m i  —j  mj=mi  a 

Summing  from  j  =  0,  •  •  • ,  i  gives  the  closed  form  ~  ip .  ANCESTORS^  =  (i,  s{i/}),p)  executes 
in  ~  i^-1  steps  because  there  is  that  many  ways  ~  i  ancestor  vertices  could  reach  v' ,  and  only 
one  way  v'  could  reach  v ; 

(g)  The  worst  case  time  bound  is  dominated  by  the  time  spent  in  ANCESTORS,  which  can  be 
added  to  the  time  spent  in  REDUCE.  Since  REDUCE(u,p),  with  a  bound  ~  i? ,  is  called  only  a 
bounded  number  of  times,  and  REDUCE^,?),  with  a  time  bound  of  ~  ip~l ,  is  called  at  most 
~  i  times,  the  worst  case  t  ime  to  process  any  x<  is  ~  t'C  for  each  i  =  0,  ■  •  • ,  n  +  1  and  longest 
production  p; 

(h)  Summing  from  i  =  0,  •••,»+  1  gives  REC  a  general  time  bound  ~  *?+1 . 

This  bound  indicates  that  Tomita’s  algorithm  belongs  to  complexity  class  0(n3)  only  if  applied 
to  grammars  in  Chomsky  normal  form  (CNF)3  or  some  other  equally  restricted  notation.  Although 
any  context-free  grammar  can  be  automatically  converted  to  CNF  (Hopcraft  and  Ullman,  1979), 
extracting  useful  information  from  the  derivation  trees  produced  by  such  grammars  would  be  time 
consuming  at  best  (if  possible  at  all). 


3  In  CNF,  productions  can  have  one  of  two  forms,  A  — ►  BC  or  A  — ►  a;  thus,  the  length  of  the  longest 
production  is  at  most  2. 
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5.  MODIFYING  THE  ALGORITHM  FOR  N3  TIME 


In  this  section,  I  explain  how  to  turn  this  algorithm  into  an  n3  recognizer  for  context-free 
grammars  in  any  unrestricted  form.  ANCESTORS  is  the  only  function  which  forces  us  to  use  i? 
steps.  It  is  interesting  to  note,  however,  that  ANCESTORS  can  take  this  many  steps  even  though 
it  returns  at  most  ~  i  ancestor  vertices,  and  even  though  there  are  at  most  ~  i  intervening  vertices 
and  edges  between  a  vertex  in  Uj  and  its  ancestors.  This  points  out  the  fact  that  ANCESTORS 
follows  the  same  subpaths  more  than  once.  The  efficiency  of  ANCESTORS  can  be  greatly  improved 
if  this  redundancy  is  eliminated. 

The  modification  described  here  turns  ANCESTORS  into  a  table  look-up  function.  That  is, 
assume  that  there  is  a  two-dimensional  “ancestors"  table.  One  dimension  is  indexed  on  the  vertices 
in  the  graph-structured  stack,  and  the  other  is  indexed  on  integers  c  =  1  to  p,  where  p  equals 
the  greatest  p.  Each  entry  (u,c)  is  the  set  of  ancestor  vertices  a  distance  of  c  from  vertex  v. 
Then,  ANCESTORS(u,  c)  returns  the  (at  most)  ~  n  ancestor  at  (u,c)  in  ~  1  steps.  Of  course,  the 
table  must  be  filled  dynamically  during  the  recognition  process,  and  so  we  must  calculate  the  time 
expended  in  this  task. 

In  Figure  5.1,  ANCESTORS  is  defined  as  a  table  look-up  function  that  dynamically  generates 
table  entries  the  first  time  they  are  requested.  In  this  definition,  the  “ancestor”  table  is  represented 
by  changing  the  parent  field  /  of  a  vertex  v  =  (i,  s,  /}  from  a  set  of  parent  vertices  to  .an  ancestor 
field  a.  For  a  vertex  v  =  (i,  s,a ),  a  consists  of  a  set  of  tuples  (c,  lc),  such  that  lc  is  the  set  of  ancestor 
vertices  a  distance  of  c  from  v.  Thus,  the  portion  of  entries  for  each  vertex  in  the  ancestor  table  is 
associated  with  the  vertex  itself.4 

Figure  5.1  illustrates  the  necessary  modifications  made  to  the  definitions  of  Figure  4.1.  (No 
changes  are  made  to  REC.)  In  SHIFT,  [1]  adds  a  vertex  to  Ui  labeled  s.  If  such  a  vertex  does  not 
already  exist,  one  is  created  whose  ancestor  field  records  that  v  is  the  ancestor  vertex  at  a  distance 
of  1;  otherwise,  v  is  added  those  ancestors. 

In  REDUCE,  [2]  iterates  through  the  ancestors  v'  a  distance  of  p  from  v,  finding  the  appropriate 
state  s"  to  shift  to  from  the  goto  table;  [3]  adds  a  vertex  labeled  s"  to  Ui- 1  (if  no  such  vertex  already 
exists)  whose  ancestors  field  is  initialized  to  point  back  to  v' .  if  such  a  vertex  does  exist,  [4]  checks 
for  an  ambiguity,  if  there  is  no  ambiguity,  then  v'  is  merged  with  the  other  ancestors  a  distance  of  1 

4  While  this  definition  may  at  first  seem  obtuse,  it  was  adopted  to  suggest  an  implementation  that 
could  take  advantage  of  the  LISP  garbage  collector  to  dynamically  recover  table  entries  when  a  process  is 
terminated. 
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from  v" .  [5]  first  checks  that  v'  is  not  a  dummy  vertex  (as  described  in  Section  4)  created  by  [6]  in 
an  earlier  call  to  REDUCE;  [6]  checks  if  v"  has  already  been  processed.  If  so,  it  applies  REDUCE  to 


SHIFTS, s) 

[1] 

If  ~  3(i,  s,  a 

)  e  Ui, 

let  Ui  := 

Ui  U  { (*,  s,  {(1,  {*■’})})} 

else 

let  / 1  := 

/x  U  {t>}  |  (l,/i)  G  o 

REDUCE (u,p) 

[2] 

For  each  v' 

=  (/, s', ax')  €  ANCESTORS (x.,p). 

Let  s"  := 

GOTO  (s',  Dp) 

[3] 

If  ~  3u" 

II 

1 

o 

m 

i — i 

T 

let  Ui- 

X  :=  f/,-i  U  {(i  —  l.s”,  {(1,  {r'}>})} 

else 

[4] 

If  v'  £  /j 

1  3(1, /x)  €  ax". 

[5] 

if  ~  3(/,s',a2')  G  /, 

let  l 

i  :=  /i  U  {i/} 

C6] 

if  v" 

€  P. 

let  vd  { i-l,s",a2 "  -  {(l,{t’'})}) 

for  each  ‘re  p’  G  ACTI0NS(s",  x,), 

REDUCED, p) 

[7]  for  each  (c,lC2)  G  a2"  |  c>2, 

if  3(c, lCl)  G  ax" 
lot  lCl  :=  la  U  /Cj 

else 

let  ax"  :=  ax"  U  {(c,/Cj)} 

else 

do  nothing  (ambiguous) 

ANCESTORSft;  =  ( j,S,a),c ) 

[8]  If  c  =  0, 

return({«}) 

else 

If  3(c,/c)  G  a, 
return  (/c) 
else 

[9]  let  lc  :=  Uv'e/.Ki.Meo  ANCESTORS ( v' ,c  -  1) 
let  a  :=  a  U  {(c,/c)} 

return(/c) 

Fig.  5.1 — Modified  Algorithm5 

vd  (a  dummy  v ")  for  each  reduce  action  on  Xj.  After  the  application  of  REDUCE,  [7]  updates  the 
ancestor  table  stored  in  v"  to  record  entries  made  in  the  ancestor  field  a2"  of  the  dummy  vertex. 

In  ANCESTORS,  [8]  looks  up  in  a  (the  portion  of  the  ancestor  table  stored  with  r)  those  vertices 
at  a  distance  of  c;  if  an  entry  exists,  those  vertices  are  returned,  but  if  not  [9]  calls  ANCESTORS 


5  The  definition  of  REC  from  Figure  4.1  is  unchanged. 
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recursively  to  generated  those  vertices  and,  before  returning  the  generated  vertices,  records  them  in 
the  ancestor  field  of  v. 

The  question  now  becomes  how  much  time  is  spent  filling  the  ancestor  table.  For 
ANCESTORS(t>,p),  this  is  bounded  in  the  worst  case  by  ~  i2  steps,  and  for  ANCESTORS(t’d,p),  it 
is  bounded  by  ~  i  steps.  This  is  because,  in  general,  ANCESTORS(v  =  (i,  s,  a),c)  will  take  ~  i  steps 
to  execute  the  first  time  it  is  called  (one  for  each  recursive  call  to  ANCESTORSfr'.e  —  1)  (1/  6  It  and 
(l,/i)  €  a)),  plus  the  steps  in  the  recursive  call,  and  ~  1  steps  thereafter.  When  ANCESTORS(u,p) 
is  executed,  there  are  ~  i  such  “virgin”  vertices  between  v  (in  {/,)  and  its  ancestors,  and  so  this 
call  can  execute  ~  i2  steps  in  the  worst  case.  ANCESTORS^, p)  is  called  only  after  the  call  to 
ANCESTORS(r,p)  has  been  made,  and  so  ~  i  of  the  vertices  between  v'  and  the  ancestor  vertices 
have  beer  processed;  hence,  the  call  to  ANCESTORS(t/,p  —  1)  could  take  ~  i  steps  for  each  of  a 
bounded  number  of  intervening  vertices. 

Given  this,  the  upper  bound  on  the  number  of  steps  that  can  be  executed  by  the  total  calls  on 
REDUCE  for  a  given  x,  is  ~  t2.  Summing  from  i  =  0,  -  •  • ,  n  +  1  gives  ~  n3  steps  as  the  worst  case 
upper  bound  on  the  execution  time  of  the  modified  algorithm. 
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6.  SPACE  BOUNDS 


While  it  might  be  suggested  that  the  modifications  introduced  in  the  last  section  make  an 
unfavorable  trade  off  between  time  and  space,  analysis  actually  shows  that  space  efficiency  is  impaired 
by  at  most  a  constant  factor. 

The  space  complexity  of  Tomita’s  algorithm  as  it  appears  in  Section  4  is  proportional  to  n2  in 
the  worst  case  (ignoring  the  read-only  space  requirements  for  the  parse  table).  This  is  because  the 
space  requirements  of  the  algorithm  are  bounded  by  the  requirements  of  the  graph-structured  stack. 
There  are  a  bounded  number  of  vertices  in  each  U<  of  the  graph-structured  stack,  and  each  vertex 
can  have  at  most  ~  i  parents.  Summing  again  from  i  =  0,  •  •  • ,  n  +  1  gives  us  ~  n2  as  the  worst  case 
space  requirement  for  the  graph-structured  stack. 

With  the  modification  given  in  Section  5,  we  have  increased  the  space  requirements  of  the 
graph-structured  stack  by  at  most  a  constant  factor  of  n2.  This  is  because  the  modification  replaces 
the  ~  i  parents  of  a  vertex  in  Ui  with  at  most  ~  pi  entries  in  the  ancestors  field.  So,  for  a  vertex 
v  —  ( i,s,a )  G  U,,  the  ancestors  field  a  will  be  a  subset  of  tuples  (c,lc)  such  that  1  <  c  <  p  and 
|/c|  ~  i.  Summing  from  i  =  0,  •  •  • ,  n  +  1,  gives  us  ~  pv?  or  ~  n2  still  as  a  worst  case  upper  bound 
on  space. 
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7.  LESS  THAN  N3  TIME 


As  mentioned  earlier  in  the  introduction,  several  of  the  better  known  general  context-free  algo¬ 
rithms  have  been  shown  to  run  in  less  than  0(n3)  time  for  certain  subclasses  of  grammars.  Therefore, 
it  is  of  interest  to  ask  if  Tomita’s  algorithm,  as  well  as  the  modified  version  presented  here,  can  also 
recognize  some  subclasses  of  context-free  grammars  in  less  than  0(n3)  time.  In  this  section,  I  in¬ 
formally  describe  two  such  subclasses  that  can  be  recognized  in  0(n2)  and  O(n)  time,  respectively. 
The  arguments  for  their  existence  parallel  those  given  by  Earley  in  his  thesis  (Earley,  1968),  where 
they  are  formally  specified. 

Time  0(n2)  Grammars.  ANCESTORS  is  the  only  function  that  forces  us  to  use  ~  ip  steps 
in  Tomita’s  algorithm  and  ~  i 2  steps  in  the  modified  algorithm.  We  determined  that  this  could 
happen  when  a  ancestor  vertex  v'  from  Uj  (j  <  i)  reached  the  reducing  vertex  v  in  by  more  than 
a  single  path,  i.e. ,  the  symbols  xj  ■  ■  ■  Xj  were  derived  from  a  nonterminal  Dp  in  more  than  one  way, 
indicating  that  grammar  G  is  ambiguous.  If  G  were  unambiguous,  then  there  would  be  at  most  one 
path  from  a  given  v'  to  v.  This  means  that  the  bounded  calls  to  ANCESTORS(n,p)  can  take  at 
most  ~  i  steps,  and  that  ANCESTORS^  ,p)  can  take  at  most  a  bounded  number  of  steps.  The  first 
observation  is  due  to  the  fact  that  there  are  ~  i  ancestor  vertices  that  can  be  reached  in  only  one 
way.  Similarly,  the  second  is  due  to  the  fact  that  if  ANCESTORS^, p)  took  ~  i  steps,  returning 
~  i  ancestors,  and  was  called  ~  i  times,  then  some  ancestor  vertices  must  have  shifted  into  (/,  in 
more  than  one  way,  which  would  be  a  contradiction,  meaning  grammar  G  must  be  ambiguous. 

So,  if  the  grammar  is  unambiguous,  then  the  total  time  spent  in  REDUCE  for  any  x<  is  ~  i 
and  the  worst  case  time  bound  for  the  Tomita’s  algorithm  is  0(n2).  A  similar  result  is  true  for  the 
modified  algorithm. 

Time  0(n)  Grammars.  In  his  thesis,  Earley  points  out  that  “  . . .  for  some  grammars  the  number 
of  states  in  a  state  set  can  grow  indefinitely  with  the  length  of  the  string  being  recognized.  For  some 
others  there  is  a  fixed  bound  on  the  size  of  any  state  set.  We  call  the  latter  grammars  bounded  state 
grammars."  While  Earley’s  “states”  have  a  different  meaning  than  states  in  Tomita’s  algorithm, 
a  similar  phenomena  occurs,  i.e.,  for  the  bounded  state  grammars  there  is  a  fixed  bound  on  the 
number  of  parents  any  vertex  can  have. 

In  Tomita’s  algorithm,  bounded  state  grammars  can  be  recognized  in  time  O(n)  for  the  following 
reason.  No  vertex  can  have  more  than  a  bounded  number  of  ancestors  (if  otherwise,  then  ~  i  vertices 
could  be  added  to  the  parents  of  some  vertex  in  [/,  ,  proving  by  contradiction  that  the  grammar  is  not 
bounded  state).  This  then  means  that  the  ANCESTORS  function  can  execute  in  a  bounded  number 
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of  steps.  Likewise,  REDUCE  can  only  be  called  a  bounded  number  of  times.  Summing  over  the  x, 
gives  us  an  upper  bound  ~  n.  Again,  a  similar  result  is  true  for  the  modified  algorithm.  Interestingly 
enough,  Earley  states  that  almost  all  LR(fc)  grammars  are  bounded  state,  as  well,  which  suggests 
that  Tomita’s  algorithm,  given  fc-symbol  look  ahead,  should  perform  with  little  loss  of  efficiency  as 
compared  to  a  standard  LR(ifc)  algorithm  when  the  grammar  is  “close"  to  LR(fc).  Earley  also  points 
out  that  not  all  bounded  state  grammars  are  unambiguous;  thus,  there  are  non-LR(lt)  grammars  for 
which  Tomita’s  algorithm  can  perform  with  LR(ib)  efficiency. 
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8.  CONCLUSION 


The  results  in  this  paper  support  in  part  Tomita’s  claim  of  efficiency  for  his  algorithm.  With 
the  modification  introduced  here,  Tomita’s  algorithm  is  shown  to  be  in  the  same  complexity  class 
as  existing  general  context-free  algorithms.  These  results  also  give  support  to  his  claim  that  his 
algorithm  should  run  with  near  LR(fc)  efficiency  for  near  LR(fc)  grammars. 

Because  the  algorithm  is  a  bottom-up  parser  that  takes  advantage  of  the  multi-action  LR  parse 
table,  it  avoids  the  extraneous  computation  associated  with  the  goal  expansion  found  in  other  general 
algorithms.  This  could  eliminate  a  factor  of  n2  steps,  which  in  practical  applications  could  result  in 
significant  gains  in  performance.  The  variation  on  Tomita’s  algorithm  described  in  Section  4  and 
the  modified  algorithm  have  been  implemented  in  LISP;  the  former  implementation  is  distributed 
with  the  ROSIE  programming  language. 

Although  the  modified  algorithm  is  theoretically  more  efficient,  the  difference  in  practical  terms 
is  less  clear.  Typical  sentences  in  a  typical  grammar  (even  a  grammar  as  ambiguous  as  ROSIE’s)  will 
probably  never  realize  the  0(ni>+1)  time  bound.  The  variation  on  Tomita’s  algorithm  distributed 
with  ROSIE  was  implemented  before  the  development  of  the  modified  algorithm.  Minimal  empirical 
results  show  that  this  implementation  runs  on  the  order  of  n  log  n  for  sentences  ranging  in  length 
from  one  to  several  thousand  symbols.  It  is  unlikely  that  a  similar  implementation  of  the  modified 
algorithm  would  perform  substantially  better.  Thus,  rather  than  accepting  the  modified  algorithm 
as  an  “improved”  algorithm  for  practical  purposes,  it  should  instead  be  treated  as  merely  a  proof  that 
Tomita’s  algorithm  can  belong  to  the  0(n3)  class,  illustrating  its  place  in  regards  to  the  best-known 
general  context-free  algorithms. 
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